High bandwidth memory system with distributed request broadcasting masters

ABSTRACT

A system comprises a processor and a plurality of memory units. The processor is coupled to each of the plurality of memory units by a plurality of network connections. The processor includes a plurality of processing elements arranged in a two-dimensional array and a corresponding two-dimensional communication network communicatively connecting each of the plurality of processing elements to other processing elements on same axes of the two-dimensional array. Each processing element that is located along a diagonal of the two-dimensional array is configured as a request broadcasting master for a respective group of processing elements located along a same axis of the two-dimensional array.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/712,253 entitled HIGH BANDWIDTH MEMORY SYSTEM WITH DISTRIBUTEDREQUEST BROADCASTING MASTERS filed Dec. 12, 2019 which is incorporatedherein by reference for all purposes.

BACKGROUND OF THE INVENTION

A whole class of complex artificial intelligence problems can be solvedusing neural networks. Since these problems are often computationallyand data intensive, hardware solutions are often beneficial forimproving the performance of neural networks. Solving these complexproblems typically requires processing large amounts of data. Due tothese data requirements, the performance of memory-based operations iscritical. Processing large amounts of data often involves acorresponding large number of memory transfers. It is a technicalchallenge to create a hardware platform for solving neural networkswhile achieving memory access performance and efficiency requirements.Therefore, there exists a need for a hardware platform that minimizesthe expense of memory transfers to effectively perform memory operationsneeded for neural network processing.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is a block diagram illustrating an embodiment of a system forsolving artificial intelligence problems using a neural network.

FIG. 2 is a block diagram illustrating an embodiment of a system forsolving artificial intelligence problems using a neural network.

FIG. 3 is a block diagram illustrating an embodiment of a system forsolving artificial intelligence problems using a neural network.

FIG. 4 is a block diagram illustrating an embodiment of a system forsolving artificial intelligence problems using a neural network.

FIG. 5 is a block diagram illustrating an embodiment of a processingelement for solving artificial intelligence problems using a neuralnetwork.

FIG. 6 is a flow chart illustrating an embodiment of a process forperforming memory access.

FIG. 7 is a flow chart illustrating an embodiment of a process forresponding to memory data requests.

FIG. 8 is a flow chart illustrating an embodiment of a process forperforming memory access.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

A high bandwidth memory system utilizing request broadcasting masters isdisclosed. To increase bandwidth to memory, a processor system iscommunicatively connected to multiple memory units. In some embodiments,the memory units are arranged surrounding a processing component. Forexample, the processing component can be arranged in a central locationrelative to the multiple memory units, which may include separate north,east, south, and west memory units. The processing component can be aprocessor with multiple processing elements, where each processingelement includes its own control logic and matrix compute engine. Theprocessing elements are arranged in a two-dimensional array, such as an8×8 matrix of processing elements. Other appropriate numbers ofprocessing elements can be utilized as well. The processing elements ofthe processor can work together in parallel by applying a neural networkto solve complex artificial intelligence problems. A network connectsthe processing elements to one another and to the memory units. Forexample, an 8×8 matrix of processing elements (64 total processingelements) is connected by an 8×8 network, such as a network-on-chipsubsystem. Each processing element can send data to other processingelements and/or access one of the memory units via the network. In someembodiments, a request broadcasting master is designated for a group ofprocessing elements. The request broadcasting master serves as a masterto manage the memory access requests of the processing elements in thegroup. For example, a request broadcasting master is designated for eachrow of processing elements, where the group is every processing elementin the row. Alternatively, the group can be every processor in a columnand a request broadcasting master is designated for each column ofprocessing elements. Any memory request from a processor in a group ismanaged by the request broadcasting master of the group. In variousembodiments, each processing element in the group sends its memoryrequests to its request broadcasting master. The request broadcastingmaster merges all memory requests from the processing elements of thegroup into a compressed memory access request. The compressed memoryaccess request reduces the number of total memory access requests buteach memory access request may be for more data. The requestbroadcasting master directs the merged memory access request to memoryunits. In some embodiments, the merged memory access request isbroadcasted to all memory units. For example, the merged memory accessrequest is transmitted along the row and column of the network subsystemto memory units on the north, east, south, and west side of theprocessing component.

In various embodiments, memory access requests to memory units aredirected to memory units only by request broadcasting masters and not byeach processing element. By designating specific request broadcastingmasters and compressing memory requests into fewer (but larger)requests, the total number of memory requests on the network at any onetime is minimized. The reduction in messages significantly improves theefficiency of memory transfers in part by minimizing network collisions.In various embodiments, each memory unit responds to its responsibleportion of a memory access request. For example, a north memory unitresponds to only the portion of the request the north memory unit isresponsible for. Similarly, east, south, and west memory units respondto only the portion of the request that they are respectivelyresponsible for. In various embodiments, the requested data addressesfor a single memory access request are distributed across the differentmemory units. The distribution may be performed using a dynamicallyprogrammable distribution scheme. By spreading data across multiplememory units using a dynamically programmable distribution scheme, forexample, based on workload, memory utilization and efficiency isincreased and processing elements with different workloads can avoidoperating in lockstep with one another.

In some embodiments, request broadcasting masters for each group ofprocessing elements are arranged offset from one another in the networksubsystem. For example, each request broadcasting master is arranged tominimize network overlap with the other request broadcasting masters andis located at a different (row, column) position in the network array orgrid. In some embodiments, the request broadcasting masters can beplaced along the diagonal of the network grid. For example, for an 8×8network, request broadcasting masters can be placed along eitherdiagonal. For a diagonal traversing from the upper left to lower right,the upper leftmost request broadcasting master transmits and receivesmemory requests to and from memory units using the top row and leftcolumn of the network. Similarly, the lower rightmost requestbroadcasting master transmits and receives memory requests to and frommemory units using the bottom row and right column of the network. Eachrequest broadcasting master along the diagonal has a dedicated columnand row for providing memory access requests to and for receiving memoryaccess responses from the different memory units. Once a memory accessresponse is received, the request broadcasting master can provide theresponse to the appropriate requesting processing element of the group.In various embodiments, the request broadcasting master and theprocessing elements of the group implement a group protocol tocoordinate the merging of memory access requests and the receiving ofresponses. For example, the request broadcasting master and eachprocessing element may perform a handshake to coordinate memory accessrequests and responses.

In some embodiments, a processor system includes a plurality of memoryunits and a processor coupled to each of the plurality of memory unitsby a plurality of network connections. For example, a processor orprocessing component is surrounded by memory units on four sides and hasmultiple network connections to each memory unit. The processor includesa plurality of processing elements arranged in a two-dimensional array,such as a two-dimensional matrix or grid of processing elements. In someembodiments, the two-dimensional array is not a strict rectangular gridbut another appropriate ordered arrangement of processing elements. Theprocessor includes a corresponding two-dimensional communication networkcommunicatively connecting each of the plurality of processing elementsto other processing elements on same axes of the two-dimensional array.For example, a network-on-chip subsystem connects the processingelements arranged in the same column and those arranged in the same row.In some embodiments, each processing element of the plurality ofprocessing elements located along a diagonal of the two-dimensionalarray is configured as a request broadcasting master for a respectivegroup of processing elements of the plurality of processing elementslocated along a same axis of the two-dimensional array. For example,processing elements arranged along the same row (or column) form a groupof processing elements. Each group has a designated request broadcastingmaster. The request broadcasting masters are located along a diagonal ofthe processing elements array. For example, no two request broadcastingmasters share the same row or the same column.

In some embodiments, each processing element of the processing componentcan be configured with a distribution scheme to scatter data across theavailable memory units. The distribution scheme is dynamicallyprogrammable such that different processing elements can apply the sameor different distribution schemes. For example, in various embodiments,each processing element can be programmed using a processor instructionto dynamically configure the distribution scheme for that processingelement. In various embodiments, processing elements sharing the sameworkload can be programmatically configured to utilize the samedistribution scheme and processing elements with different workloads canbe programmatically configured to utilize different distributionschemes. Different distribution schemes help to prevent multipleprocessing elements from working in lockstep with one another. Byvarying the distribution schemes, the memory units are more efficientlyutilized and memory performance is increased. In some embodiments, thesize of the memory unit access units is also configurable. For example,the size of the memory unit access units can be programmaticallyconfigured via a processor instruction. Each processing element can readand/or write data to each memory unit via a request broadcasting masterusing a configurable access unit-sized group. Moreover, memory accessoperations can span multiple access units and reference data distributedacross multiple memory units. In various embodiments, each memory accessrequest is broadcasted to all memory units and each memory unit returnspartial responses that are combined to fulfill the broadcasted request.

In some embodiments, a processor system comprises a plurality of memoryunits and a processor coupled to the plurality of memory units. Forexample, a processor system includes a processor communicativelyconnected to multiple memory units. In some embodiments, the memoryunits are arranged on all sides of the processor to help minimizelatency from the processor to each memory unit. Each of the plurality ofmemory units includes a request processing unit and a plurality ofmemory banks. For example, a request processing unit receives memoryaccess requests, such as read and/or write requests, and determineswhether and how to process the requests. The request processing unit candetermine whether a portion of the memory access request can be servedby the memory unit and its corresponding memory banks. For example, therequest processing unit can decompose a memory access request intopartial requests and determine what subset of the partial requests maybe served from the corresponding memory banks of the memory unit. Invarious embodiments, each memory unit can include multiple memory banksto increase the memory size of a memory unit. For example, a memory unitcan include 4, 8, 16, 32, or another appropriate number of memory banks.In some embodiments, the processor includes a plurality of processingelements. For example, the processor is a processing component thatincludes a group of processing elements. The processing elements may bearranged in a matrix, such as an 8×8 array of processing elements. Theprocessor also includes a communication network communicativelyconnecting the plurality of processing elements to the plurality ofmemory units. For example, a communication network such as anetwork-on-chip subsystem and/or network interfaces/busescommunicatively connect each processing element to each memory unit. Insome embodiments, each processing element of the plurality of processingelements includes a control logic unit and a matrix compute engine. Forexample, a first processing element of the plurality of processingelements includes a control logic for controlling the first processingelement and a matrix compute engine for computing matrix operations. Thecontrol logic is configured to access data from the plurality of memoryunits using a dynamically programmable distribution scheme. For example,the control logic is configured using a processor instruction to utilizea specific distribution scheme or pattern. The scheme may be based onthe processing element workload or another appropriate configuration.The distribution scheme determines the mapping of memory addressesspecific to the processing element to memory locations of the memoryunits.

FIG. 1 is a block diagram illustrating an embodiment of a system forsolving artificial intelligence problems using a neural network. In theexample shown, the system 100 is a hardware platform that includesprocessing component 101 and memory units 111, 121, 131, and 141.Processing component 101 is communicatively connected to memory units111, 121, 131, and 141 via a network connection such as networkconnections 151. Network connections 151 communicatively connectsprocessing component 101 to north memory unit 111. Similar networkconnections (not labeled) connect processing component 101 to memoryunit 121, 131, and 141. Processing component 101 is connected to and cancommunicate with each of memory unit 111, 121, 131, and 141simultaneously. The memory units 111, 121, 131, and 141 are positionedaround processing component 101 at north, east, south, and westpositions but other layouts are appropriate. By positioning memory units111, 121, 131, and 141 around processing component 101, memory units111, 121, 131, and 141 can be accessed simultaneously by processingcomponent 101 and/or multiple connections may be used by processingcomponent 101 to communicate with different memory units 111, 121, 131,and 141 in parallel. In the example shown, system 100 includes fourmemory units that surround a processing component but fewer or morememory units may be utilized as appropriate.

In some embodiments, processing component 101 is a processor thatincludes one or more processing elements (not shown). Each processingelement may include at least a matrix compute engine for performingmatrix operations. The processing elements may be furthercommunicatively connected using a communication network and/or bus suchas a network-on-chip subsystem. Data for performing neural networkoperations may be retrieved from and written to memory units such asmemory units 111, 121, 131, and 141 located around processing component101. For example, using a network-on-chip subsystem, memory accessoperations can be directed to memory, including memory units 111, 121,131, and 141, from a processing element of processing component 101 viaa request broadcasting master. The request broadcasting master mergesmemory requests from a group of processing elements. Responses to therequests are received by the request broadcasting master and transmittedto the original requesting processing element(s) of the group. In someembodiments, each processing element can be assigned a particularworkload and each workload may be associated with a particular set ofdata stored in memory. For example, the set of data for a workload mayinclude activation and/or filter matrix data. In various embodiments,the data is associated with large neural network matrices and mayinclude hundreds or more matrix elements. The relevant data may bestored across different regions of memory units 111, 121, 131, and 141.In some embodiments, the data is stored in access unit-sized groupsdistributed across memory units 111, 121, 131, and 141 based on adynamically programmable distribution scheme.

In some embodiments, processing component 101 is communicativelyconnected to memory units 111, 121, 131, and 141 via a set ofnetwork/bus connections such as network connections 151 on the northside of processing component 101. For example, a network-on-chipsubsystem of processing component 101 includes a network array or gridthat connects an array of processing elements of processing component101. In some embodiments, each column of the network grid is connectedto memory unit 111 on the north side of processing component 101 and tomemory unit 131 on the south side of processing component 101.Similarly, each row of the network grid is connected to memory unit 121on the east side of processing component 101 and to memory unit 141 onthe west side of processing component 101. In some embodiments, thenumber of external network connections on each side of processingcomponent 101 matches the number of input/output connections of eachmemory unit. For example, network connections 151 may include 32 networkconnections from processing component 101 to memory unit 111, oneconnection of network connections 151 matching each north-side externalnetwork connection of processing component 101 to an input/outputconnection of memory unit 111.

In some embodiments, the data stored in memory units 111, 121, 131,and/or 141 may be accessed by workload or another appropriateidentifier. For example, a workload identifier may be used to determinehow to distribute and retrieve data across the different availablememory units. In various embodiments, different workloads are programmedto distribute their corresponding workload data across available memoryunits using different distribution schemes. For example, each workloadcan be dynamically programmed to use a different distribution scheme. Invarious embodiments, a distribution scheme uses a configurable orderedpattern for accessing memory units. Instead of using a predefineddistribution for all workloads, a processing element can be dynamicallyprogrammed to distribute data differently from other processingelements. This allows for better utilization and efficiency of thememory units. In various embodiments, the data associated with a memoryaccess operation may reside in one or more different memory units. Forexample, a memory read request may be served by data located in memoryunits 111, 121, and 131. A different memory request may be served bydata in memory units 121, 131, and 141. In some embodiments, a hashfunction, such as a programmable hash function, is used to determine thememory layout scheme or access order pattern for a particular workloador identifier. For example, a memory read request for one processingelement may access memory units using a repeating ordered patternstarting with memory unit 111 followed by memory unit 121, memory unit131, and memory unit 141. A memory read request for a differentprocessing element may use a different programmable repeating orderedpattern starting with memory unit 141 followed by memory unit 121,memory unit 131, and memory unit 111. Since data is distributed acrossdifferent memory units, a memory request can trigger one or more partialresponses from different memory units that each respond to a portion ofthe memory request. Once all partial responses have been received by aprocessing element, the memory request is complete.

In some embodiments, a memory access operation, such as a write or readmemory access operation, can be split into multiple partial accessrequests. The memory access operation may be a merged memory accessoperation that includes (and compresses) requests from multipleprocessing elements. In some embodiments, the memory access operation isdecomposed or unrolled into one or more partial access requests bymemory units 111, 121, 131 and/or 141. Based on the memory rangerequested, a memory unit determines whether it contains the requesteddata. In some embodiments, the memory request is processed by a requestprocessing unit of the memory unit (not shown). For example, in someembodiments, a memory request is broadcasted to all memory units and isprocessed by the corresponding request processing unit of each memoryunit. Each request processing unit analyzes the request and differentrequest processing units respond to different portions of the memoryrequest. For example, a request processing unit responds only torequests for data or memory addresses associated with its memory unit.In the event a memory access request can be served by a particularmemory unit, the associated request processing unit can retrieve therelevant data from (or write the relevant data to) the associated memoryunit. Memory access requests that cannot be served by the particularmemory unit can be ignored and will be handled by the correspondingappropriate memory unit. In some embodiments, each memory unit containsmultiple memory banks and the request processing unit can direct thepartial memory access request to the appropriate memory bank of thememory unit.

In some embodiments, the size of a data access unit used by each memoryunit is programmable. For example, memory units can be programmed to usea 128 byte or another appropriately sized access unit such that everynew group of 128 bytes (or another appropriate access unit size) isstored on a different memory unit based on the programmable distributionscheme. This allows the data to be written across different memory unitsusing programmable sized access units. For example, the first accessunit of data is written to a first memory unit, the second access unitof data is written to a second memory unit, and so forth, as determinedby the ordering of the distribution scheme. Once all memory units havebeen utilized, the next memory unit cycles back to the first memoryunit. In various embodiments, the order of the memory units can also beprogrammable and may be determined using a hashing function. Forexample, each workload may utilize a different distribution order foraccessing memory units based on the outcome of the hashing function.

FIG. 2 is a block diagram illustrating an embodiment of a system forsolving artificial intelligence problems using a neural network. In theexample shown, the system 200 is a hardware platform that includesprocessing component 201 and memory units 211, 221, 231, and 241.Processing component 201 is communicatively connected to memory units211, 221, 231, and 241 via multiple network connections. Processingcomponent 201 includes an array of processing elements (each labeled“PE”), including processing elements 203, 205, 207, and 209. Processingelements in bold, such as processing elements 203, 205, and 207, areprocessing elements designated as request broadcasting masters. Someprocessing elements are shown with dashed lines to reflect theadditional number of processing elements to fill out an array ofprocessing elements and are processing elements that do not function asrequest broadcasting masters. The ellipses between processing elementsindicate additional rows or columns of processing elements. The size ofthe array of processing elements can vary as appropriate. In someembodiments, processing component 201 includes an 8×8, 16×16, 32×32, oranother appropriate sized array of processing elements. In the exampleshown, network subsystem 251 communicatively connects the processingelements of processing component 201 to one another and to memory units211, 221, 231, and 241. On each side of processing component 201,multiple network connections connect processing component 201 to amemory unit. In some embodiments, system 200 is system 100 of FIG. 1 ,processing component 201 is processing component 101 of FIG. 1 , andmemory units 211, 221, 231, and 241 are memory units 111, 121, 131, and141, respectively, of FIG. 1 .

In some embodiments, memory units are located on each of the four sidesof processing component 201. For example, memory unit 211 is located onthe north side of processing component 201, memory unit 221 is locatedon the east side of processing component 201, memory unit 231 is locatedon the south side of processing component 201, and memory unit 241 islocated on the west side of processing component 201. Each memory unitincludes multiple memory banks. In the example shown, each memory unitis depicted with four memory banks but each can be configured withanother appropriate number of memory banks. The ellipses between memorybanks of the same memory unit indicate optional additional memory banks.In some embodiments, each memory unit is communicatively connected to aside of processing component 201 by multiple networkconnections/interfaces. Memory requests and corresponding responses foreach memory unit can be transmitted via any of the network connectionsof a memory unit. By utilizing multiple network connections for eachmemory bank, a memory unit can receive memory requests from and respondwith memory responses to multiple processing elements in parallel. Themultiple network connections for each memory unit increase the overallmemory bandwidth.

In some embodiments, network subsystem 251 is a network-on-chipsubsystem that connects each of the processing elements of processingcomponent 201 to one another. Network subsystem 251 is an array or gridnetwork with communication lines arranged along the columns and rowscorresponding to the processing element array. Network subsystem 251 isfurther communicatively connected to memory units 211, 221, 231, and241. The network communication lines for each column and row connect toinput/output interfaces of the memory units. For example, the networkcommunication lines for each column connect to memory unit 211 on thenorth side of processing component 201 and to memory unit 231 on thesouth side of processing component 201. Similarly, the networkcommunication lines for each row connect to memory unit 221 on the eastside of processing component 201 and to memory unit 241 on the west sideof processing component 201. In various embodiments, the number ofcolumn communication lines corresponds to the number of input/outputinterfaces of memory units 211 and 231. The number of row communicationlines corresponds to the number of input/output interfaces of memoryunits 221 and 241. Although system 200 is depicted with four memoryunits, in some embodiments, the total memory units may be a subset ofthe memory units shown in FIG. 2 . For example, memory units may only belocated on two or three sides of processing component 201 andcorresponding network communication lines of network subsystem 251 mayonly include network interfaces to access memory units whereappropriate. A system configured with only three memory banks may havenetwork communication lines/interfaces to connect to memory units ononly three sides of the processing component.

In some embodiments, the processing elements are arranged into groups byrows (or columns). For example, each row of processing elements isdesignated as a group. One of the processing elements in the group isdesignated as the request broadcasting master. For example, processingelement 203 is the request broadcasting master for the processingelement group corresponding to the first row of processing elements.Similarly, processing element 205 is the request broadcasting master forthe processing element group corresponding to the second row ofprocessing elements. For the last row of processing elements, processingelement 207 is the request broadcasting master for the group. Thedesignated request broadcasting masters are aligned along the diagonalof the array of processing elements. In the example shown, requestbroadcasting masters 203, 205, and 207 are located along a diagonal,traversing from the upper left to lower right, of the processingelements array. In some embodiments, the request broadcasting mastersare located along a different diagonal, such as the diagonal traversingfrom the lower left to the upper right of the processing elements array.In some embodiments, the groups are defined by columns instead of rowsand a single request broadcasting master is designated for each columngroup of processing elements. Memory requests and responses from aprocessing element are directed using the request broadcasting master.Requests are transmitted to the request broadcasting master where theyare forwarded to memory units. Responses from memory units are receivedby the request broadcasting master and transmitted to the originatingprocessing element.

In some embodiments, each processing element of an array forwards memoryrequests to the request broadcasting master of the group, such asrequest broadcasting masters 203, 205, and 207. Processing elements thatare not request broadcasting masters do not directly communicate withmemory units. For example, processing element 209 forwards all of itsmemory access requests to request broadcasting master 205. Each requestbroadcasting master then broadcasts the memory request to all memoryunits. For example, a memory access request is broadcasted using networksubsystem 251 to memory units 211, 221, 231, and 241. The request isbroadcasted to north and south memory units, memory units 211 and 231,respectively, via a network connection traversing along the columndirection of network subsystem 251. The request is also broadcasted toeast and west memory units, memory units 221 and 241, respectively, viaa network connection traversing along the row direction of networksubsystem 251. Each request broadcasting master broadcasts memoryrequests in all directions towards all memory units using networksubsystem 251. In various embodiments, the memory access requests may beread and/or write requests. Since the request broadcasting masters arelocated along the diagonal of the processing elements array, theirrespective broadcasts have minimal overlap and likelihood of collisionwith one another. Each request broadcasting master communicates with amemory unit using a different network interface. In some embodiments,all request broadcasting masters can communicate with the same memoryunit in parallel. In response to a memory access request, memory unitsprovide responses that are transmitted back to the request broadcastingmaster using the same route but in a reverse direction. By using thesame route, responses directed to different request broadcasting mastershave minimal overlap and likelihood of collision with one another. Sinceeach request broadcasting master has its own dedicated route on networksubsystem 251 for broadcasting requests and receiving responses, thelikelihood of network collisions is significantly reduced.

In some embodiments, each of the request broadcasting masters of a groupreceive and then merge the memory access requests from the processingelement of its group. For example, a memory request received by requestbroadcasting master 205 from processing element 209 is merged by requestbroadcasting master 205 with one or more memory requests from processingelements of the same group. By merging memory requests, the total numberof broadcasted requests are reduced, further reducing network trafficand the likelihood of collisions. Since data may be distributed acrossmultiple memory units, multiple memory units may service the memoryrequests by sending partial responses addressing only the portions ofthe memory request each memory unit is responsible for.

In some embodiments, memory units 211, 221, 231, and 241 each receivebroadcasted memory access requests from a request broadcasting master ofprocessing component 201. The memory access requests may be read and/orwrite requests. Each of memory units 211, 221, 231, and 241 decomposesthe memory access request to determine whether it can be served,potentially partially, by one of its memory banks. Although four memorybanks are shown in FIG. 2 for each memory unit, in various embodiments,memory unit 211, 221, 231, and 241 can include fewer or many more memorybanks such as 8, 16, 32, 64, or another appropriate number of memorybanks. In some embodiments, whether a memory unit can service a portionof a memory request is determined using a hashing function to implementa dynamically programmable distribution scheme. For example, the hashingfunction may utilize a workload identifier of a processing element todistribute data across memory units and banks based on a processingelement's workload. In some embodiments, the hashing function inspects aset of bits, such as two or more bits, of a memory address associatedwith the memory access request.

In some embodiments, the memory read/write size, such as the size of amemory access unit, can be programmable. For example, memory reads canbe programmed to be 64 bytes, 128 bytes, or another appropriate accessunit size. Each memory unit can determine the appropriate bytes to readand/or write by analyzing each incoming memory access request. In theevent a request can be served by the memory unit, such as memory unit211, 221, 231, or 241, a memory request response will be returned toprocessing component 201 and the appropriate requesting processingelement(s) via the corresponding request broadcasting master. In someembodiments, prepared responses may include data read from a memorybank. The response may be a partial response that fulfills only aportion of the original memory access request. Additional partialresponses may be fulfilled by other memory units responsible formanaging the corresponding memory address ranges. For example, a largememory read request broadcasted to all memory units may be fulfilled bymultiple partial responses supplied by multiple memory units. In someembodiments, each partial response includes an identifier such as asequence identifier that may be used to order the partial responses. Forexample, partial responses may not be received in order and anidentifier is used to sort the partial responses and build a completeresponse from multiple partial responses.

FIG. 3 is a block diagram illustrating an embodiment of a system forsolving artificial intelligence problems using a neural network. In theexample shown, the system 300 is a hardware platform that includesprocessing component 301 and memory units 311, 321, 331, and 341.Processing component 301 is communicatively connected to memory units311, 321, 331, and 341 via multiple network connections. Processingcomponent 301 includes an array of processing elements (each labeled“PE”), including processing elements 303 and 305. Processing elements inbold, such as processing elements 303 and 305, are processing elementsdesignated as request broadcasting masters. Some processing elements areshown with dashed lines to reflect the additional number of processingelements to fill out an array of processing elements and are processingelements that do not function as request broadcasting masters. Theellipses between processing elements indicate additional rows or columnsof processing elements. The size of the array of processing elements canvary as appropriate. In some embodiments, processing component 301includes an 8×8, 16×16, 32×32, or another appropriate sized array orgrid of processing elements. In the example shown, network subsystem 351communicatively connects the processing elements of processing component301 to one another and to memory units 311, 321, 331, and 341. On eachside of processing component 301, multiple network connections connectprocessing component 301 to a memory unit. Processing element andrequest broadcasting master 303 broadcasts a memory access request tomemory units 311, 321, 331, and 341 using network route 353 on networksubsystem 351. Processing element and request broadcasting master 305broadcasts a memory access request to memory units 311, 321, 331, and341 using network route 355 on network subsystem 351. In someembodiments, system 300 is system 100 of FIG. 1 , processing component301 is processing component 101 of FIG. 1 , and memory units 311, 321,331, and 341 are memory units 111, 121, 131, and 141, respectively, ofFIG. 1 . In some embodiments, system 300 is system 200 of FIG. 2 ,processing component 301 is processing component 201 of FIG. 2 , memoryunits 311, 321, 331, and 341 are memory units 211, 221, 231, and 241,respectively, of FIG. 2 , and network subsystem 351 is network subsystem251 of FIG. 2 .

In some embodiments, processing elements of the same processing elementgroup transmit memory requests to a request broadcasting master, such asrequest broadcasting master 303 or 305. The request broadcasting mastermerges the received requests from the processing elements of its group.In some embodiments, the merged memory request is a compressed versionof the original individual memory requests. For example, the compressedor merged memory request may reduce duplicative requests, that is,requests from different processing elements of the same group thatoverlap in requested data. In various embodiments, each request mayinclude an identifier associated with a configurable distribution schemefor distributing data across memory units. Once merged, the memoryrequest is broadcasted over network subsystem 351 using a network route.For example, request broadcasting master 303 utilizes network route 353to broadcast memory access requests for memory units 311, 321, 331, and341. A request is broadcasted along a dedicated column and row ofnetwork subsystem 351. Similarly, request broadcasting master 305utilizes network route 355 to broadcast memory access requests formemory units 311, 321, 331, and 341. The points of collision for networkroute 353 and network route 355 are minimal since request broadcastingmaster 303 and request broadcasting master 305 are located at differentrow and column locations from one another. By locating the requestbroadcasting masters along a diagonal of the processing elements array,collisions between requests and responses are significantly reduced. Theresponses from memory units 311, 321, 331, and 341 utilize the samenetwork route to respond to requests but travel in the reversedirection. For example, memory unit 311 utilizes the same networkinterface associated with network route 353 to respond to requests fromwhich a memory request is received to direct a response back to requestbroadcasting master 303. As another example, memory unit 311 utilizesthe same network interface associated with network route 355 to respondto requests from which a memory request is received to direct a responseback to request broadcasting master 305. Since the return routes forresponses from memory unit 311 to request broadcasting master 303 andrequest broadcasting master 305 do not overlap, network collisions areminimized and the effective memory bandwidth is increased.

FIG. 4 is a block diagram illustrating an embodiment of a system forsolving artificial intelligence problems using a neural network. In theexample shown, the system 400 is a hardware platform that includesprocessing component 401 and memory units 411, 421, and 441. A fourthmemory unit is not depicted in FIG. 4 but is located on the south sideof processing component 401. Each memory unit includes a requestprocessing unit and multiple memory banks. For example, memory unit 411includes request processing unit 413 and memory banks such as memorybank 415. Each memory unit, such as memory unit 411, may be configuredwith additional (or fewer) memory banks than shown. The internalcomponents of memory units 421 and 441 are not shown but resemble memoryunit 411.

In some embodiments, processing component 401 is communicativelyconnected to memory units including memory units 411, 421, and 441 viamultiple network connections. Processing component 401 includes an arrayof processing elements (each labeled “PE”), including processingelements 403, 405, 461, 463, and 465. Processing elements in bold, suchas processing elements 403 and 405, are processing elements designatedas request broadcasting masters. Some processing elements are shown withdashed lines to reflect the additional number of processing elements tofill out an array of processing elements and are processing elementsthat do not function as request broadcasting masters. The ellipsesbetween processing elements indicate additional rows or columns ofprocessing elements. The size of the array of processing elements canvary as appropriate. In some embodiments, processing component 401includes an 8×8, 16×16, 32×32, or another appropriate sized array ofprocessing elements. The outline of processing component 401 is dashedto depict that only a portion of processing component 401 is shown,emphasizing the relationship between processing component 401 and memoryunit 411. In various embodiments, the relationship between processingcomponent 401 and memory unit 411 is similar to how processing component401 interacts with the remaining memory units.

In the example shown, network subsystem 451 communicatively connects theprocessing elements of processing component 401 to one another and tomemory units 411, 421, and 441. On each side of processing component401, multiple network connections connect processing component 401 to amemory unit. Processing element and request broadcasting master 403broadcasts a memory access request to memory units, including memoryunits 411, 421, and 441, using network subsystem 451. Processing elementand request broadcasting master 405 broadcasts a memory access requestto memory units, including memory units 411, 421, and 441, using networkroute 455 on network subsystem 451. Responses from memory unit 411traverse network route 471, using the same network interface the requestis received from for memory unit 411. Similarly, responses from memoryunit 421 traverse network route 473, using the same network interfacethe request is received from for memory unit 421 and responses frommemory unit 441 traverse network route 477, using the same networkinterface the request is received from for memory unit 441. Responsesfrom a memory unit (not shown) on the south side of processing component401 traverse network route 475, using the same network interface therequest is received from.

In some embodiments, processing elements of the same processing elementgroup transmit memory requests to a request broadcasting master, such asrequest broadcasting master 403 or 405. The request broadcasting mastermerges the received requests from the processing elements of its group.For example, processing elements 405, 461, 463, and 465 are a row ofprocessing elements and form a processing element group. Processingelement 405 functions as the request broadcasting master of the group.As described with respect to FIG. 3 , request broadcasting master 405broadcasts a memory access request to all available memory units usingnetwork route 455 on behalf of all processing elements of the group.North side memory unit 411 and a south side memory unit (not shown)receive the broadcasted request via a north/south network communicationline of network subsystem 451. East side memory unit 421 and west sidememory unit 441 receive the broadcasted request via an east/west networkcommunication line of network subsystem 451. The four directions thebroadcast is transmitted are shown by network route 455. In variousembodiments, the broadcasted request is a merged memory request thatcompresses a number of individual requests originating from processingelements of the same group, such as one or more of processing elements405, 461, 463, and/or 465, and is broadcasted by request broadcastmaster and processing element 405.

In some embodiments, memory unit 411 includes request processing unit413 and multiple memory banks such as memory bank 415. Requestprocessing unit 413 receives broadcasted memory access requests fromrequest broadcasting master and processing element 405 of processingcomponent 401 via network route 455. The memory access requests may beread and/or write requests. Request processing unit 413 decomposes thememory access request to determine whether it can be served, potentiallypartially, by one of the memory banks of memory unit 411. Although fourmemory banks are shown in FIG. 4 , in various embodiments, memory unit411 can include fewer or many more memory banks (as represented by theellipses) such as 8, 16, 32, 64, or another appropriate number of memorybanks. In some embodiments, request processing unit 413 directs memoryaccess requests to the appropriate memory bank(s) of memory unit 411.For example, based on the memory address of the request, requestprocessing unit 413 determines the appropriate memory bank(s) to access.In some embodiments, two or more memory banks of memory unit 411 may beaccessed for a single memory access request. The memory banks may bedetermined based on a hashing function. For example, the hashingfunction may utilize a workload identifier of the processing elementassociated with the original memory access request. In some embodiments,the hashing function inspects a set of bits, such as two or more bits,of a memory address associated with the memory access request. A memoryaccess response is prepared by request processing unit 413 andtransmitted to request broadcasting master 405 via network route 471.Request broadcasting master 405 transmits the response to the processingelement from where the request initially originates. In someembodiments, each memory unit prepares and transmits partial responsesto correspond to portions of the broadcasted memory request that eachmemory unit is responsible for. A completed response can be constructedfrom partial responses. In some embodiments, the construction of thecomplete response is performed by the request broadcasting master, suchas request broadcasting master 405. In some embodiments, theconstruction of the complete response is performed by the originalprocessing element using forwarded partial responses from the requestbroadcasting master. Similar to memory unit 411, the other memory unitsof system 400, including memory units 421, 441, and other memory unitsnot shown, operate in a similar manner to respond to broadcasted memoryrequests.

In some embodiments, system 400 is system 100 of FIG. 1 , processingcomponent 401 is processing component 101 of FIG. 1 , and memory units411, 421, and 441 are memory units 111, 121, and 141, respectively, ofFIG. 1 . In some embodiments, system 400 is system 200 of FIG. 2 ,processing component 401 is processing component 201 of FIG. 2 , memoryunits 411, 421, and 441 are memory units 211, 221, and 241,respectively, of FIG. 2 , and network subsystem 451 is network subsystem251 of FIG. 2 . In some embodiments, system 400 is system 300 of FIG. 3, processing component 401 is processing component 301 of FIG. 3 ,memory units 411, 421, and 441 are memory units 311, 321, and 341,respectively, of FIG. 3 , network subsystem 451 is network subsystem 351of FIG. 3 , request broadcasting master 405 is request broadcastingmaster 305 of FIG. 3 , and network route 455 is network route 355 ofFIG. 3 .

FIG. 5 is a block diagram illustrating an embodiment of a processingelement for solving artificial intelligence problems using a neuralnetwork. In the example shown, processing element 500 includes controllogic 501, memory management unit 503, local store memory 505, networkinterface 507, and matrix compute engine 509. In various embodiments,one or more processing elements can work together on the same data setor workload to solve an artificial intelligence program using a largeworking data set. In some embodiments, processing element 500 is aprocessing element of processing component 101 of FIG. 1 , processingcomponent 201 of FIG. 2 , processing component 301 of FIG. 3 , and/orprocessing component 401 of FIG. 4 . In some embodiments, processingelement 500 is processing element 203, 205, 207, 209, and/or anotherprocessing element of FIG. 2 , processing element 303, 305, and/oranother processing element of FIG. 3 , and/or processing element 403,405, 461, 463, 465, and/or another processing element of FIG. 4 . Insome embodiments, processing element 500 includes functionality as arequest broadcasting master, such as request broadcasting masters 203,205, and/or 207 of FIG. 2, 303 and/or 305 of FIG. 3 , and/or 403 and/or405 of FIG. 4 .

In some embodiments, control logic 501 is a control logic unit fordirecting the functionality of processing element 500 and may be used tointerface with the components of processing element 500 such as memorymanagement unit 503, local store memory 505, network interface 507, andmatrix compute engine 509. In some embodiments, control logic 501 mayrespond to processor instructions used to apply a neural network to anartificial intelligence problem. For example, control logic 501 can beused to initiate reading and/or writing of data from memory via networkinterface 507 in response to a processing instruction. In someembodiments, control logic 501 is used to load and prepare operatingarguments for matrix compute engine 509. For example, control logic 501can prepare matrix operands for computing a convolution operation. Insome embodiments, control logic 501 is used to help process partialresponses to a memory data request.

In some embodiments, processing element 500 functions as a requestbroadcasting master and control logic 501 implements the requestbroadcasting master functionality described herein. For example, controllogic 501 is used to merge memory access requests from processingelements of the same group into a compressed or merged memory accessrequest that is broadcasted to memory units. Control logic 501 is usedto receive and process partial responses to the broadcasted request. Insome embodiments, the request broadcasting master constructs a completeresponse from received partial responses before transmitting thecomplete response to the original requesting processing element. In someembodiments, the request broadcasting master forwards partial responsesto the appropriate processing elements and each processing elementitself constructs a complete response from received partial responses.In some embodiments, the request broadcasting master functionality isimplemented in a component separate from control logic 501, such as arequest broadcasting master (not shown).

In some embodiments, memory management unit 503 is used to manage memoryrelated functionality of processing element 500. For example, memorymanagement unit 503 may be used to program the access unit size used forreading data from and/or writing data to memory units such as memoryunits 111, 121, 131, and/or 141 of FIG. 1 . In some embodiments, a largememory read is divided into access unit-sized groups and one of theavailable memory units is responsible for servicing each memory group.Distributing the data across memory units in access unit-sized groupsallows memory to be accessed much more efficiently and significantlyimproves memory utilization. In some embodiments, memory management unit503 is used to configure a hashing mechanism for distributing the dataacross different memory units. For example, memory management unit 503can manage configurations associated with a programmable hashingmechanism. In some embodiments, memory management unit 503 is part ofcontrol logic 501. Instead of using a fixed distribution pattern for allmemory access operations, the programmable hashing mechanism allows thedistribution pattern to be configurable. For example, differentprocessing element workloads can use different distribution patterns. Asone example, one workload can be configured to write to memory unitsusing a north, east, south, west pattern while another workload can beconfigured to write to the memory units using a south, north, east, westpattern. In various embodiments, the distribution scheme is dynamic andcan be dynamically programmed via control logic 501 and memorymanagement unit 503. Memory management unit 503 is used to help maplocal memory addresses to different memory access unit-sized groupsfound in different memory units.

In some embodiments, local store memory 505 is a memory scratchpad forstoring data such as data related to neural network operations. Localstore memory 505 may be used for storing data retrieved via partialresponses to memory access requests. Partial responses and theassociated data may be gathered and stored in local store memory 505 tobuild a complete response. In some embodiments, local store memory 505is made up of registers for fast read and write access. In variousembodiments, one or more components of processing element 500, such asmatrix compute engine 509, can access local store memory 505. Forexample, matrix input data operands and/or output data results can bestored in local store memory 505.

In some embodiments, local store memory 505 is used by a processingelement acting as a request broadcasting master to store memory requestsfrom processing elements of the same group. For example, memory requestsmay be temporarily stored to create a merged memory request that can bebroadcasted to available memory units. The merged memory requestcompresses multiple requests from one or more processing elements of thesame group into a single memory request that requests the datareferenced by the individual requests. In some embodiments, the mergingoperation utilizes local store memory 505. In various embodiments, localstore memory 505 is used to direct responses received in response to themerged memory request back to the original processing element from whicha memory access request originates. For example, the address of anoriginating processing element and the requested memory address rangeare stored in local store memory 505.

In some embodiments, network interface 507 is used to interface with anetwork subsystem such as a network-on-chip system for networkcommunication. In some embodiments, the network subsystem that networkinterface 507 communicates with is network subsystem 251 of FIG. 2 ,network subsystem 351 of FIG. 3 , and/or network subsystem 451 of FIG. 4. Memory access requests from and to processing element 500 such as readand write requests are transmitted via network interface 507. Forexample, memory access requests can be transmitted via network interface507 to a request broadcasting master. Similarly, in some embodiments, aprocessing element functioning as a request broadcasting master receivesmemory requests from processing elements of the same group, broadcaststhe merged memory access requests to memory units, and receives partialresponses from memory units via network interface 507.

In some embodiments, matrix compute engine 509 is a hardware matrixprocessor unit for performing matrix operations including operationsrelated to convolution operations. For example, matrix compute engine509 may be a dot product engine for performing dot product operations.In some embodiments, the convolution operations supported includedepthwise, groupwise, normal, regular, pointwise, and/orthree-dimensional convolutions, among others. For example, matrixcompute engine 509 may receive a first input matrix such as a subset ofa large image represented as a three-dimensional matrix. The first inputmatrix may have the dimensions height×width×channel (HWC),channel×height×width (CHW), or another appropriate layout format. Matrixcompute engine 509 may also receive a second input matrix such as afilter, kernel, or weights, etc. to apply to the first input matrix.Matrix compute engine 509 can be used to perform a convolution operationusing the two input matrices to determine a resulting output matrix. Insome embodiments, matrix compute engine 509 may include input and/oroutput buffers for loading input data matrices and writing out a resultdata matrix. The data used by matrix compute engine 509 may be read fromand/or written to local store memory 505 and/or external memory such asmemory units 111, 121, 131, and/or 141 of FIG. 1 .

FIG. 6 is a flow chart illustrating an embodiment of a process forperforming memory access. For example, an artificial intelligenceproblem is solved by applying a neural network using data associatedwith the problem and the neural network. The data is read from andwritten to memory such as memory units 111, 121, 131, and/or 141 of FIG.1 by a processing element such as processing elements 203, 205, 207,and/or 209 of FIG. 2 via a request broadcasting master such as requestbroadcasting master and processing elements 203, 205, and/or 207 of FIG.2 . In some embodiments, the process of FIG. 6 is performed by requestbroadcasting master 203, 205, and/or 207 of FIG. 2 , requestbroadcasting master 303 and/or 305 of FIG. 3 , and/or requestbroadcasting master 403 and/or 405 of FIG. 4 . In some embodiments, theprocess of FIG. 6 is performed by processing element 500 of FIG. 5 whenfunctioning as a request broadcasting master. Using the process of FIG.6 , data elements stored in memory can be distributed across multiplememory units to improve the utilization of memory and the efficiency ofmemory access operations.

At 601, memory access data requests are received. For example, memoryaccess data requests are received at a request broadcasting master fromone or more processing elements of the same group. In some embodiments,the memory access requests may be read or write requests. For example, aread request may specify a base address and a size. In some embodiments,the request includes an identifier such as a workload identifier that isused to determine which memory units are responsible for which portionsof the requested data. In various embodiments, the transmission of arequest from a processing element to a request broadcasting masterincludes implementing a network protocol that may include negotiating aconnection between the processing element and the request broadcastingmaster. The negotiating the network connection may include performing ahandshake, for example, related to setting up the network connection. Insome embodiments, the network connection is used at 607 to transmit aresponse from memory to the originating processing element via therequest broadcasting master.

In some embodiments, the identifier included in the memory accessrequest is determined by initializing the processing element. Forexample, a particular memory access distribution scheme is using aprocessor instruction such as an instruction directed to a particularprocessing element. The distribution scheme may be associated with aparticular workload such as a particular artificial intelligence problemand neural network. In some embodiments, the initialization includessetting a workload identifier. For example, a workload identifier can beused to configure how data is distributed across multiple memory units.The workload identifier may be a parameter to a processor memorymanagement instruction. Each workload can use a different distributionscheme to improve the utilization and efficiency of memory. Processingelements working on the same dataset or workload can utilize the sameworkload identifier to share data. By scattering data across memoryunits using different distribution patterns, such as differentdistribution patterns for each workload, the data stored in memory ismore efficiently distributed across all available memory. In someembodiments, the memory initialization includes configuring the memoryaccess unit size. For example, a memory access unit, such as 128 bytes,256 bytes, etc., can be configured such that data is written to eachmemory unit in access unit-sized groups. Larger or smaller access unitscan be used as appropriate. Data within an access unit group is storedin the same memory unit. In some embodiments, the access unit size isconfigurable using a programmable instruction to a processor orprocessing element.

In some embodiments, processing element initialization includesconfiguring or programming a hashing mechanism for distributing dataacross memory units. For example, a hashing mechanism can utilize a seedto configure the distribution scheme. In some embodiments, the seed isbased on specifying a group of bits from a memory address to determinewhich memory unit is assigned to a particular access unit of data. Forexample, the hashing mechanism may specify two bits of the memoryaddress, such as two upper bits, and perform a bitwise operation on thespecified bits to map an access unit to a memory unit. In someembodiments, the bitwise operation utilizes an XOR operation. In someembodiments, the hashing mechanism can be programmatically configured.For example, a processing element can be configured to utilize aspecified hashing function and/or be configured to utilize certainparameters for the hashing function.

At 603, memory data requests are merged. Using the memory access datarequests received at 601 from one or more processing elements, a requestbroadcasting master merges the requests into a merged memory datarequest. By merging multiple requests, the number of total requests tomemory units is reduced. Merging requests reduces the amount of networktraffic and significantly reduces the number of potential collisions. Insome embodiments, the merged data request includes identifierinformation for determining the dynamically programmable distributionscheme each memory request utilizes. In some embodiments, two or moreprocessing elements may request the same or overlapping data. At 603,the duplicative requests are merged into a single request and therequesting processing elements can rely on the same response.

At 605, a merged memory data request is broadcasted to all memory units.For example, the merged memory access data request created at 603 isbroadcasted by a request broadcasting master to all memory units. Therequest traverses the network subsystem using a network route thatincludes only the communication lines along the row and column of therequest broadcasting master to reach available memory units. Requestbroadcasting masters are located along the diagonal of the processingelements array and therefore each have a unique pair of column and rowcommunication lines. The broadcasted requests for each requestbroadcasting master have minimal network overlap with broadcasts fromother request broadcasting masters and their respective responses.

In some embodiments, the network subsystem used for the broadcast is anetwork-on-chip subsystem such as network subsystem 251 of FIG. 2, 351of FIG. 3 , and/or 451 of FIG. 4 . Examples of network routes abroadcast traverses include network route 353 and 355 of FIG. 3 andnetwork route 455 of FIG. 4 . In some embodiments, four memory units,such as a north, east, south, and west memory unit, surround aprocessing component such as processing component 101 of FIG. 1 . In theexample, all four memory units, such as memory units 111, 121, 131, and141, receive the broadcasted memory data request. In some embodiments,the data request is for a large amount of data and includes dataspanning multiple access units. The request can be constructed toreference at least a base memory address and a size argument todetermine how much data is requested starting at the base memoryaddress. In some embodiments, multiple base addresses and sizes aremerged together into the merged memory request. Other memory referencingschemes may be appropriate as well. In some embodiments, the broadcastedmemory request also includes mapping information corresponding to thedistribution scheme. For example, receiving memory units can use themapping information to determine the programmatically configured hashingmechanism and/or hashing mechanism parameter(s) used by the processingelement initiating the request. As another example, the mappinginformation may also include the programmatically configured access unitsize. In various embodiments, the memory data request may be provided tomemory units for reading data or writing data.

At 607, memory data responses are received and transmitted. For example,partial memory data responses are received from memory units at therequest broadcasting master. Each partial memory data responsecorresponds to a requested portion of the memory access requestbroadcasted at 605. For example, two or more partial memory dataresponses are received from two or more different memory units. Sincethe memory request spans multiple access units, multiple memory unitscan respond, each providing a partial response corresponding todifferent access units, to complete the entire request. Each memory unitcreates one or more partial responses associated with the one or moreaccess units it is responsible for. For example, data associated with amemory request can be spread across three memory units. Each of thethree memory units responds with a partial memory data response. At 607,the partial memory responses are received. In some embodiments, eachresponse includes an identifier such as a sequence identifier fororganizing the partial responses into a complete response.

In some embodiments, once a partial response is received, the responseis provided to the original processing element that generated the memoryaccess data request received at 601. In various embodiments, theresponses are partial responses and require multiple partial responsesto construct a complete response. Each partial response is forwarded bythe request broadcasting master to the originating processing elementfor that processing element to construct a complete response. In variousembodiments, the network connection utilized for transmitting theresponses was created at 601 to receive the original memory request.

In some embodiments, the partial responses are stored at the requestbroadcasting master and a complete response is constructed from thepartial responses by the request broadcasting master. Once a completeresponse is constructed, the request broadcasting master provides thecompleted response to the originating processing element. For example, arequest broadcasting master stores the necessary data of receivedpartial responses from memory units until all the data needed toconstruct a full response has been received. Once a complete responsecan be constructed, the request broadcasting master transmits thecompleted response to the originating processing element. Instead offorwarding partial responses, the request broadcasting master forwardsonly a completed response. In some embodiments, having the requestbroadcasting master construct a complete response minimizes the networktraffic among the group of processing elements since only completeresponses are forwarded and not every partial response.

FIG. 7 is a flow chart illustrating an embodiment of a process forresponding to memory data requests. For example, a memory unit utilizesthe process of FIG. 7 to respond to a broadcasted memory data request.The memory unit decomposes the memory request and determines whichaccess units the memory unit is responsible for and then prepares andsends one or more partial responses for the access units managed by thememory unit. In some embodiments, the process of FIG. 7 is performed bymemory units 111, 121, 131, and/or 141 of FIG. 1 and/or the memory unitsof FIGS. 2-4 . In some embodiments, the process of FIG. 7 is performedin response to a memory data request broadcasted at 605 of FIG. 6 . Insome embodiments, the response prepared using the process of FIG. 7 isreceived by a request broadcasting master, such as request broadcastingmaster 203, 205, and/or 207 of FIG. 2 , request broadcasting master 303and/or 305 of FIG. 3 , and/or request broadcasting master 403 and/or 405of FIG. 4 .

At 701, a memory data request is received. For example, a memory datarequest spanning multiple access units is received. Some of the accessunits are associated with the memory unit and others may be associatedwith a different memory unit. In various embodiments, multiple memoryunits may receive the same memory data request as a broadcasted memorydata request. In some embodiments, the memory data request includes abase address and a size parameter to determine the address rangerequested. The memory data request may also include mapping informationto determine the hashing mechanism and/or hashing mechanism parameter(s)used for the particular memory distribution scheme of the memory accessrequest. In some embodiments, memory data request mapping informationincludes the access unit size.

At 703, the memory data request is decomposed into partial requests. Forexample, a request spanning multiple access units is split into partialrequests. In some embodiments, the decomposing is performed by unrollingthe memory data request into partial requests based on a configuredaccess unit size. For example, a memory data request spanning threeaccess units is decomposed into three partial requests, one for eachaccess unit. As another example, in some embodiments, each memory unitis responsible for multiple access units. For example, in a scenariowith a memory data request spanning 32 memory access units that areevenly distributed across four memory units, each memory unit isresponsible for eight partial requests. Each partial request correspondsto a memory access unit of data managed by the memory unit.

At 705, data for associated partial requests is accessed. For example,data of access units that match a partial request are retrieved from (orwritten to) memory banks of the memory unit. In some embodiments, amemory unit may have multiple memory banks and the data of thecorresponding partial requests is stored in one or more memory banks ofthe memory unit. In some embodiments, the data accessed is in responseto a partial request decomposed from a larger request spanning multipleaccess units. In the case of a memory access read operation, thecorresponding data is read from memory banks of the memory unit in theevent the partial request matches to the memory unit. Similarly, in thecase of a memory access write operation, the corresponding data iswritten to memory banks of the memory unit in the event the partialrequest matches to the memory unit.

In some embodiments, a partial request is mapped with a correspondingmemory unit based on a programmable distribution scheme. For example,different workloads can distribute data to memory units using differentdistribution schemes configured using a hashing mechanism. In variousembodiments, at 705, the hashing mechanism for the configureddistribution scheme is used to determine whether the memory unitreceiving the memory data request is responsible for the partialrequest. In the event the memory unit manages that particular addressrange of the partial request, the corresponding data is retrieved (orwritten). Otherwise, the partial request is ignored and will be handledby the correct memory unit responsible for that address range.

At 707, partial memory data responses are prepared and sent. Forexample, data read from memory units is packaged into responsesassociated with partial requests. In some embodiments, the responseprepared corresponding to a read operation is a partial memory dataresponse since it includes only a portion of the requested data. Invarious embodiments, each partial response includes an identifier suchas a sequence identifier for ordering the partial responses into acomplete response. The identifier of each partial memory data responsecan be utilized by a request broadcasting master to order a set ofpartial responses that are received out of order. The response istransmitted to a request broadcasting master for one or more processingelements to receive. In some embodiments, the response is anacknowledgement that a request corresponding to the write operation iscomplete.

FIG. 8 is a flow chart illustrating an embodiment of a process forperforming memory access. For example, a request broadcasting masterutilizes the process of FIG. 8 to gather data corresponding to a memorydata request for a read operation. In some embodiments, a requestbroadcasting master, such as request broadcasting master 203, 205,and/or 207 of FIG. 2 , request broadcasting master 303 and/or 305 ofFIG. 3 , and/or request broadcasting master 403 and/or 405 of FIG. 4 ,receives partial memory data responses from multiple memory units suchas memory units 111, 121, 131, and/or 141 of FIG. 1 and/or the memoryunits of FIGS. 2-4 . In some embodiments, the process of FIG. 8 isperformed by the processing element from which a memory access requestoriginates and not by a request broadcasting master. For example, therequest broadcasting master forwards received partial responses to theoriginating processing element for the originating processing element toconstruct a complete response from the partial responses. In someembodiments, the process of FIG. 8 is performed in response to a memorydata request broadcasted at 605 of FIG. 6 and/or in response to partialmemory data responses sent using the process of FIG. 7 . In someembodiments, the process of FIG. 8 is performed at 607 of FIG. 6 togather partial responses from a variety of memory units.

At 801, a data memory partial response is received. For example, apartial response to a data memory request sent from a memory unit isreceived. In various embodiments, the response includes data that is oneor more access units in size from the same memory unit. In someembodiments, the response includes identifier information such as asequence identifier that can be used to order the partial responsereceived relative to other partial responses.

At 803, the data memory partial response is identified. For example,using an identifier included in the received partial response, the datamemory partial response is identified relative to the original datamemory request. For example, a request may be decomposed or unrolledinto five partial requests. The partial response is identified at 803 todetermine which of the five partial responses it corresponds to. In someembodiments, the identification is performed by inspecting an identifiersuch as a sequence identifier. The identification result can be used todetermine the ordering of the partial response relative to other partialresponses and to reconstruct a complete response from the set ofreceived partial responses.

At 805, a data memory partial response is stored in local memory. Forexample, data read from memory is extracted from the data payload of apartial response and stored in local memory. In some embodiments, atemporary buffer sized for the requested data is allocated from localmemory to construct a complete response from partial responses. Sincepartial responses may be received out of order relative to theircorresponding memory addresses, the data from the partial response isstored in the allocated buffer at a corresponding location based on therelationship of the partial response to the original requested data. Forexample, a buffer sized for five partial responses is allocated and thedata from the received partial response is written to a correspondingaddress location in the buffer regardless of when the partial responseis received. In some embodiments, each partial response is an accessunit-sized response or a multiple of an access unit. In variousembodiments, the local memory is local memory store 505 of FIG. 5 .Using the temporary buffer, a completed data memory response can bereconstructed from partial responses.

At 807, a determination is made whether the response is complete. Forexample, a response is complete once all partial responses that arerequired to construct a completed response are received. In the eventthe response is complete, processing proceeds to 809. In the event theresponse is not complete, processing loops back to 801 to receive anadditional partial response.

At 809, memory data request processing is completed. For example, thedata corresponding to a complete response is made available foradditional computation such as matrix computation. In some embodiments,the data associated with the completed response is located in localmemory such as a local memory store of the processing element. Thecompleted response may be used as input to a matrix compute engine ofthe processing element and/or distributed to other processing elements.For example, other processing elements associated with the requestbroadcasting master receive their corresponding requested data. In someembodiments, the completed response corresponds to data describing aneural network or activation data associated with an artificialintelligence problem.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A system, comprising: a plurality of memoryunits; and a processor coupled to each of the plurality of memory unitsby a plurality of network connections, wherein the processor includes aplurality of processing elements arranged in a two-dimensional array anda corresponding two-dimensional communication network communicativelyconnecting each of the plurality of processing elements to otherprocessing elements on same axes of the two-dimensional array, andwherein each processing element of the plurality of processing elementslocated along a diagonal of the two-dimensional array is configured as arequest broadcasting master for a respective group of processingelements of the plurality of processing elements located along a sameaxis of the two-dimensional array.
 2. The system of claim 1, whereineach processing element of the plurality of processing elements includesa matrix compute engine, a network interface, and a control logic. 3.The system of claim 2, wherein the control logic is configured toprovide a memory request to the request broadcasting master for therespective group of processing elements and to access data from theplurality of memory units using a dynamically programmable distributionscheme.
 4. The system of claim 1, wherein the request broadcastingmaster for the respective group of processing elements is configured toreceive a plurality of memory requests from the plurality of processingelements of the respective group.
 5. The system of claim 4, wherein therequest broadcasting master is configured to merge the plurality ofmemory requests into a compressed memory request.
 6. The system of claim5, wherein the request broadcasting master is configured to broadcastthe compressed memory request to the plurality of memory units.
 7. Thesystem of claim 6, wherein the request broadcasting master is configuredto receive partial memory responses in response to the broadcastedcompressed memory request from the plurality of memory units.
 8. Thesystem of claim 6, wherein the broadcasted compressed memory requestreferences data stored in each of the plurality of memory units.
 9. Thesystem of claim 6, wherein each of the plurality of memory units isconfigured to decompose the broadcasted compressed memory request into acorresponding plurality of partial requests.
 10. The system of claim 9,wherein each of the plurality of memory units is configured to determinewhether each of the corresponding plurality of partial requestscorresponds to data stored in a corresponding one of a plurality ofmemory banks associated with the corresponding memory unit.
 11. Thesystem of claim 10, wherein each of the plurality of memory units isconfigured to provide a partial response associated with a different oneof the corresponding plurality of partial requests.
 12. The system ofclaim 11, wherein the partial response includes a corresponding sequenceidentifier that orders the partial response among a plurality of partialresponses.
 13. The system of claim 6, wherein the each requestbroadcasting master is configured to receive partial responses, combinethe partial responses to generate a complete response to the broadcastedcompressed memory request, and provide the complete response to aprocessing element of the respective group of processing elements. 14.The system of claim 6, wherein the each request broadcasting master isconfigured to receive partial responses, match each of the partialresponses to a processing element of the respective group of processingelements, and forward each of the matched partial responses to thecorresponding matched processing element.
 15. The system of claim 1,wherein the each request broadcasting master located along the diagonalof the two-dimensional array is configured to provide memory requests toand receive responses from the plurality of memory units using adifferent network connection of the plurality of network connections.16. The system of claim 1, wherein the plurality of memory unitsincludes a north memory unit, an east memory unit, a south memory unit,and a west memory unit.
 17. A method comprising: receiving a firstmemory request associated with a first processing element of a firstprocessing element group of a plurality of processing element groups,wherein each processing element group of the plurality of processingelement groups is located on a different row of a two-dimensional arrayof processing elements; receiving a second memory request associatedwith a second processing element of the first processing element group;merging the first memory request and the second memory request into acompressed memory request; broadcasting the compressed memory request toa plurality of memory units; and receiving from the plurality of memoryunits a plurality of partial responses associated with the compressedmemory request.
 18. The method of claim 17, further comprising:combining the plurality of partial responses to create a first completeresponse to the first memory request and a second complete response tothe second memory request; providing the first complete response to thefirst processing element; and providing the second complete response tothe second processing element.
 19. The method of claim 17, furthercomprising: matching a first set of partial responses of the pluralityof partial responses with the first memory request; matching a secondset of partial responses of the plurality of partial responses with thesecond memory request; providing the first set of partial responses tothe first processing element; and providing the second set of partialresponses to the second processing element.
 20. A system, comprising: aplurality of memory units, wherein at least one of the plurality ofmemory units is configured to decompose a broadcasted compressed memoryrequest into a corresponding plurality of partial requests; and aprocessor coupled to each of the plurality of memory units by aplurality of network connections, wherein the processor includes aplurality of processing elements arranged in a two-dimensional array anda corresponding two-dimensional communication network communicativelyconnecting each of the plurality of processing elements to otherprocessing elements on same axes of the two-dimensional array, andwherein each processing element of the plurality of processing elementslocated along a diagonal of the two-dimensional array is configured as arequest broadcasting master for a respective group of processingelements of the plurality of processing elements located along a sameaxis of the two-dimensional array.